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Abstract 

Presently,  AT&T  synchronizes  its  national  networks  using  analog  signals  derived 
from  a  cesium  clock  ensemble  having  a  precision  of  a  few  parts  in  10-12.  These  analog 
signals  have  well  served  the  need  for  precise  frequency  in  what  was  predominantly 
an  analog  network.  However,  the  AT&T  network  is  rapidly  being  transitioned  to  a 
digital  network  which  needs  precise  time  rather  than  frequency.  As  a  result,  several 
alternatives  were  considered  and  the  one  chosen  was  a  new  system  based  on  Global 
Positioning  System  (GPS). 

The  GPS  based  system,  termed  a  Primary  Reference  Clock  or  PRC,  employs  a 
GPS  receiver  providing  long  term  timing  accuracy  along  with  duplicated  disciplined 
rubidium  oscillators  providing  short  term  (ie.  one  day)  stability.  Under  computer 
control,  these  three  elements  are  verified  against  each  other  and  against  identical  PRC 
systems  in  other  parts  of  the  network.  The  PRC  system  produces  a  digital  signal 
used  to  synchronize  a  master  clock  in  the  node  which  in  turn  produces  a  signal  that 
is  (1)  used  to  synchronize  all  other  clocks  in  the  node,  and  (2)  distributed  on  a  digital 
basis  to  all  neighboring  nodes.  By  use  of  monitoring  equipment  at  each  PRC,  the 
performance  of  the  master  clock  at  every  AT&T  network  node  will  be  verified  against 
the  GPS  signal  at  each  PRC.  For  the  first  time,  a  national  telecommunications  network 
will  be  monitored  and  verified  to  be  performing  to  a  precision  approaching  Universal 
Coordinated  Time  (UTC). 


INTRODUCTION 

Synchronization  of  telecommunication  networks  is  becoming  increasingly  important  as  the  evolution  of 
digital  switching  and  transmission  facilities  becomes  the  norm  in  today’s  national  networks.  The  lack  of 
adequate  synchronization  for  these  networks  would  impair  many  of  the  information  signals  important 
to  the  functioning  of  industry  and  government.  To  meet  the  need  for  high  quality  synchronization, 
AT&T  has  developed  a  new  verifiable  synchronization  network. 

This  paper  describes  this  new  synchronization  network  focusing  on  the  new  AT&T  Primary  Ref¬ 
erence  Clock  (hereafter  referred  to  as  the  PRC),  its  verification  methodology  and  capabilities,  and 
the  inter-node  and  intra-node  methodology  for  distributing  its  timing  reference  to  nodes  and  systems 
throughout  AT&T’s  digital  and  remaining  analog  networks.  The  PRC  employs  a  GPS  receiver  provid¬ 
ing  long  term  timing  accuracy  along  with  duplicated  disciplined  rubidium  oscillators  providing  short 
term  (ie.  one  day)  stability.  Under  computer  control,  these  three  elements  are  verified  against  each 
other  and  against  identical  PRC  systems  in  other  parts  of  the  network.  The  PRC  system  produces 
a  digital  signal  used  to  synchronize  a  co-located  master  clock  in  the  node,  which  in  turn  produces  a 
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signal  that  is:  (1)  used  to  synchronize  all  other  clocks  in  the  node,  and  (2)  distributed  on  a  digital 
basis  to  all  neighboring  nodes.  By  use  of  monitoring  equipment  at  each  PRC  location,  the  performance 
of  the  master  clocks  at  neighboring  AT&T  network  nodes  are  verified  against  the  PRC  signal.  For 
the  first  time,  a  national  telecommunications  network  is  monitored  and  verified  to  be  performing  to  a 
precision  approaching  Universal  Coordinated  Time  (UTC). 

BACKGROUND 

Presently,  AT&T  synchronizes  its  own  national  telecommunications  network,  as  well  as  providing 
synchronization  signals  to  most  of  the  Regional  Bell  Operating  Companies  and  many  independent 
telephone  companies,  using  analog  signals  derived  from  a  cesium  clock  ensemble  [1],  This  ensemble, 
known  as  the  Basic  Synchronization  Reference  Frequency  (  BSRF  )  is  located  in  Hillsboro,  MO.,  and 
has  a  long  term  accuracy  of  a  few  parts  in  10-12.  The  2048  KHz  reference  frequency  from  the  BSRF 
is  distributed  on  analog  radio  and  coax  systems  as  illustrated  in  Figure  1. 

When  BSRF  was  installed,  AT&T  used  frequency  division  multiplexing  in  which  even  the  most 
critical  equipment  could  tolerate  a  10-9  frequency  offset.  This  frequency  accuracy  can  be  easily  ob¬ 
tained  from  the  BSRF  distribution  system  even  with  transmission  impairments  such  as  dropouts, 
protections  switching,  and  phase  hits.  The  analog  signals  used  to  carry  the  BSRF  reference  fre¬ 
quency  have  well  served  the  need  for  precise  frequency  in  what  was  predominantly  a  long  haul  analog 
telecommunications  network. 

Today,  however,  AT&T  is  rapidly  being  transitioned  to  an  all  digital  network  in  which  time  division 
transmission  and  switching  functions  are  critical  [2].  In  general,  synchronization  of  networks  using 
time  division  systems  encompasses  bit,  frame  and  network  timing  issues  [3]. 

The  bit  timing  level  of  synchronization  deals  with  physical  layer  timing  issues  such  as:  clock 
insertion  and  recovery,  transmission  line  jitter,  sampling  windows  in  eye  patterns,  and  ones  density. 
These  issues  are  addressed  in  the  fundamental  design  of  the  transport  system  and  by  adhering  to 
engineering  rules  such  as  repeater  spacing  and  maximum  number  of  repeater  spans. 

The  frame  timing  level  of  synchronization  deals  with  identifying  groupings  of  bits  (time  slots) 
which  can  be  associated  with  a  particular  user.  A  framing  signal  encoded  in  the  digital  transmission 
system  typically  accomplishes  this  function.  When  time  slots  within  a  digital  stream  need  to  be 
processed  separately,  a  frame  alignment  process  is  performed  on  an  incoming  stream.  Once  framing 
is  determined,  each  time  slot  can  be  identified  and  stored  for  processing.  The  main  synchronization 
issues  at  the  frame  level  of  synchronization  are  reframe  time  and  framing  loss  detection. 

The  network  timing  level  deals  with  processing  of  time  slots.  Under  ideal  conditions,  time  slots 
are  sent  from  the  source  node  at  a  perfectly  uniform  rate,  delivered  over  the  transmission  system  with 
a  fixed  delay,  and  clocked  into  the  receiving  node  buffer.  The  receive  node  will  then  read  the  time 
slot  currently  in  the  store  after  a  fixed  delay.  Assuming  no  random  or  systematic  offset  of  the  receive 
node  clock  relative  to  the  sending  clock,  this  processing  will  continue  with  no  variation  in  the  buffer 
and  complete  time  slot  integrity. 

The  objective  of  network  timing,  is  to  approach  this  idealized  scenario.  Time  slot  buffers  (more 
commonly  termed  frame  alignment  buffers)  can  accommodate  125/zs  of  systematic  time  error  and  18/is 
of  random  time  error  without  slipping  time  slots.  The  maximum  long  term  systematic  frequency  de¬ 
parture  allowed  at  the  output  of  a  network  clock  is  1  x  10-u.  This  requirement  has  been  established 
for  a  number  of  years  both  domestically  [4,5]  and  internationally  [6,7],  and  is  based  on  achieving  satis¬ 
factory  slip  rates  on  an  end  to  end  basis  for  the  most  critical  services  (voiceband  data,  facsimile,  video, 
secure  voice  and  secure  data).  Provisional  short  term  requirements  allow  from  1  to  10  microseconds 
of  time  error  in  a  day  at  the  output  of  a  network  clock.  The  short  term  requirements  are  being  put  in 
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place  to  ensure  that  random  or  cyclical  daily  timing  variations  will  not  produce  daily  slips. 

The  performance  objective  of  1  X  10“ 11  for  digital  network  timing  is  two  orders  of  magnitude 
greater  than  the  frequency  accuracy  required  for  analog  transmission.  The  current  synchronization 
plan  to  time  the  digital  offices  of  AT&T  was  based  on  using  the  existing  analog  distribution  system 
and  implementing  clocks  to  receive  this  signal  and  achieve  the  1  x  10-11  recommendation. 

This  plan  was  first  implemented  in  the  early  1980’s.  Analysis  of  this  architecture  has  been  per¬ 
formed.  The  overall  results  of  this  evaluation  produced  two  key  findings.  First,  with  evaluation  and 
some  minor  redesign  of  the  BSRF  timing  signal  receiver  algorithms,  the  1  x  10-11  performance  ob¬ 
jective  is  achievable.  Second,  the  1  X  10-11  or  better  objective  is  subject  to  certain  transmission  and 
operational  impairments  levels  which  are  not  guaranteed  to  be  obtained  on  all  systems  at  all  times 
based  on  normal  network  operations.  Thus,  although  typical  performance  is  several  parts  in  10  '12,  it 
is  possible,  under  certain  condition,  for  a  node  to  degrade  to  parts  in  10-10  for  a  short  period. 

As  a  result,  AT&T  is  deploying  a  network  of  distributed  primary  reference  clocks  to  synchronize 
its  digital  network.  These  clocks  will  use  the  Global  Positioning  System  (  GPS  )  to  provide  long  term 
timing  accuracy  to  network  nodes  or  buildings  at  fourteen  locations  within  the  continental  United 
States. 

This  new  synchronization  network  will  have  rich  interconnectivity  and  the  unique  capability  of 
verification  at  several  levels.  Verification  is  the  capability  to  directly  monitor  the  timing  performance 
of  each  office  or  node.  This  capability  enables  AT&T  to  detect  and  resolve  timing  degradation  before 
it  becomes  service  impacting.  Verification  ensures  uniform  compliance  of  domestic  and  international 
timing  standards  at  all  nodes,  and  provides  a  cost  effective  tool  to  sectionalize  and  resolve  timing 
problems. 

AT&T’s  NEW  PRIMARY  REFERENCE  CLOCK  STRATEGY 

Figure  2  illustrates  the  architecture  of  the  new  verifiable  timing  distribution  plan.  In  addition  to  the 
fourteen  PRC’s  located  in  the  contiguous  United  States,  PRC’s  will  be  located  in  Hawaii  and  Puerto 
Rico.  This  new  synchronization  network  will  have  verification  capability  which  permits  the  detection 
of  timing  degradation.  The  PRC  reference  is  transmitted  to  other  nodes,  called  secondary  nodes,  which 
do  not  contain  a  PRC.  The  PRC  simultaneously  monitors  the  reference  it  receives  from  at  least  two 
other  PRC  nodes  as  well  as  secondary  nodes.  The  monitoring  of  neighboring  PRC  locations  provides 
long  term  stability  data.  By  employing  a  n-corner  hat  decomposition  [8]  of  the  PRC  level  stability 
data,  the  long  term  timing  stability  of  each  PRC  location  will  be  tracked  and  verified  to  parts  in  10-13. 
The  architecture  is  designed  so  that  secondary  nodes  are  monitored  by  two  independent  PRCs.  This 
provides  a  means  to  sectionalize  nodal  timing  instability  from  transmission  path  timing  instability. 
By  deploying  PRC’s  close  to  secondary  locations,  uniformly  stable  transmission  path  performance  is 
attained  both  for  transmitting  and  monitoring  performance.  Daily  transmission  path  stability  should 
exceed  several  parts  in  10-12. 

In  searching  for  a  source  of  precise  timing  for  a  PRC,  three  alternatives  were  studied.  The  first  was 
to  continue  to  use  a  single  cesium  ensemble  at  one  location  along  with  a  digitally  based  distribution 
system.  The  second  and  third  would  use  regionally  distributed  PRC’s  using  either  LORAN-C  or  GPS 
and  disciplined  timing  elements.  Using  a  single  cesium  ensemble  offers  some  economies,  but  does 
not  permit  nationwide  verification,  nor  does  it  offer  performance  improvement  over  today’s  network. 
Using  a  nodal  based  system  offers  a  broad  based  verification  capability.  LORAN-C  was  judged  inferior 
to  GPS  in  terms  of  time  transfer  accuracy,  inherent  operations,  and  future  capabilities,  so  GPS  was 
chosen  to  provide  AT&T  with  a  long  term  time  reference. 

Rubidium  oscillators  were  selected  over  quartz  and  cesium  as  the  disciplined  timing  elements.  Our 
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strategy  in  using  the  GPS  system  was  based  on  using  a  twenty-four  hour  observation  period  to  obtain 
a  a  time  and  frequency  error  estimate.  There  are  many  reasons  for  this  choice.  First,  a  daily  ensemble 
average  effectively  removes  the  daily  systematic  bias  errors  which  arise  in  the  time  transfer  process  of 
GPS  from  user  position  errors  and  other  sources.  Second,  ensemble  averaging  of  the  satellite  tracks 
reduces  random  error  sources  as  well  as  intentional  degradation  added  to  reduce  the  accuracy  of  the 
standard  service.  Third,  evaluation  of  ensemble  tracks  permits  the  removal  of  outlying  data  which 
can  contaminate  the  steering  process. 

For  periods  less  than  one  day,  a  set  of  duplicated  rubidium  oscillators  provide  timing  stability.  The 
stability  of  the  disciplined  rubidium  over  the  24  hour  interval  between  steering  updates  is  equivalent 
to  the  stability  of  the  steering  update  data  from  the  GPS  satellite  tracks.  For  a  24  hour  steering 
interval,  rubidium  oscillators  proved  to  provide  the  best  cost /performance  tradeoff.  Quartz  oscillators 
cannot  obtain  the  daily  stability  performance  achievable  from  GPS,  and  cesium  oscillators  provide  no 
real  stability  advantage  and  are  more  costly. 

An  AT&T  3B-2  computer  is  used  to  steer  the  rubidium  oscillators  and  to  verify  their  performance 
relative  to  one  another  and  to  the  GPS  receiver.  The  block  diagram  given  in  Figure  3  illustrates  the 
interconnectivity  of  the  PRC’s  computer,  and  indicates  the  DS1  1  inputs  from  the  other  nodes  and 
clocks  used  for  timing  monitoring.  In  the  figure,  the  rubidium  oscillators  (Rb)  are  shown  communi¬ 
cating  with  the  disciplined  controllers  which  in  turn  communicate  with  the  3B-2  monitor  computer. 
The  computer  performs  control,  performance  verification,  and  error  recovery  functions. 

The  timing  outputs  of  the  PRC  are  a  pair  of  DS1  primary  rate  digital  signals.  The  PRC  outputs 
are  used  to  provide  timing  reference  to  a  master  clock  in  a  node  containing  typically  many  clocks.  The 
master  clock,  known  as  the  Building  Integrated  Timing  Supply,  or  BITS,  provides  timing  reference  to 
all  other  clocks  in  that  node  as  illustrated  in  Figure  4.  The  BITS  provides  input  to  a  device  known  as 
a  Clock  Distribution  Unit  whose  function  is  to  provide  multiple  DS1  outputs  to  all  clocks  in  the  node 
requiring  digital  reference  (eg.  No.  4  ESS  switch,  DACS  transmission  system)  and  frequency  locked 
clocks  for  analog  frequency  division  multiplex  (FDM)  equipment. 

INTER-NODE  AND  INTRA-NODE  REFERENCE  DISTRIBU¬ 
TION 

Buildings  containing  a  PRC  are  known  as  Primary  Nodes.  Buildings  which  do  not  contain  a  PRC 
are  known  as  Secondary  Nodes  and  obtain  their  timing  reference  from  a  Primary  Node  in  most  cases, 
or  from  other  Secondary  Nodes  in  rare  cases.  Timing  to  Secondary  Nodes  is  distributed  on  a  digital 
basis  using  two  DS1  signals  which  are  routed  over  diverse  facilities  from  different  PRC’s.  These  DS1 
signals  carry  both  traffic  and  synchronization  reference  between  nodes  as  illustrated  in  Figure  5.  The 
reference  information  is  bridged  from  a  cross-connect  device  known  as  a  DSX-1  and  transmitted  to 
the  clock  of  the  system  selected  as  the  BITS.  Thereafter,  the  reference  distribution  is  the  same  as 
in  the  Primary  Node.  The  timing  is  embedded  in  the  DS1  traffic  carrying  signal  as  that  signal  is 
referenced  to  the  BITS  clock  within  the  Primary  Node  which  is  the  source  for  the  reference. 

VERIFICATION  METHODOLOGY 

As  illustrated  in  Figure  6,  there  are  four  levels  of  verification  that  are  used.  The  first  is  an  ongoing 
verification  of  GPS  performance  by  various  laboratories  such  as  the  National  Bureau  of  Standards, 

1A  DSl  is  the  primary  rate  synchronous  digital  signal  used  in  the  network.  It  is  a  1544  Kbps  signal  consisting  of  24 
64  kbps  time  slots.  Each  time  slot  typically  carries  a  single  direction  voice  circuit. 
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the  Naval  Observatory,  and  the  Network  Synchronization  Laboratory  at  AT&T  Bell  Laboratories  at 
Holmdel,  NJ.  This  high  level  verification  results  in  long  term  changes  in  the  synchronization  network 
operations  such  as  upgrading  of  steering  algorithms. 

The  second  level  of  verification  is  within  the  PRC  itself.  The  PRC  determines  the  relative  timing 
instability  or  uncertainty  associated  with  the  three  clocking  elements  in  the  system  (the  two  disciplined 
rubidium  oscillators  and  the  GPS  receiver).  A  three  corner  hat  technique  is  used  to  decompose  the 
pair-wise  instability  measurements  between  the  three  components,  and  determine  the  timing  stability 
produce  by  each  element.  This  technique  can  effectively  detect  marginal  performance  in  any  of  the 
three  clocking  elements.  This  technique  is  limited  to  detection  of  short  term  daily  timing  instability, 
since  in  the  long  term,  rubidium  timing  is  not  independent  of  the  GPS  receiver. 

The  third  level  of  verification,  called  the  Primary  Tier  (or  long  term)  verification  uses  an  N  corner 
hat  technique  to  determine  the  long  term  time  uncertainty  or  noise  of  each  of  the  Primary  Nodes  (eg. 
A,  B,  and  C  in  Figure  6). 

Finally,  the  fourth  level,  or  Secondary  Tier  Verification,  extends  verification  down  to  the  Secondary 
Nodes,  where  the  signals  returning  from  the  Secondary  Node  BITS  clocks  are  monitored  by  at  least 
two  PRC’s  to  determine  their  performance. 

PERFORMANCE 

Figure  7  illustrates  the  performance  of  the  AT&T  Primary  Reference  clock  as  compared  to  the  current 
AT&T  cesium  ensemble  and  telecommunication  synchronization  standards.  In  Figure  7,  the  time 
keeping  error  relative  to  UTC  is  shown  in  seconds  on  the  ordinate  along  with  the  observation  time  on 
the  abscissa  shown  in  seconds. 

In  case  (1),  the  slope  of  the  ANSI  and  CCITT  specification  is  the  allowed  long  term  frequency 
accuracy  of  a  PRC  of  1  x  1CT11.  The  ANSI  and  CCITT  specifications  allow  for  a  maximum  daily  time 
instability  of  3000  ns  which  is  reflected  by  the  horizontal  asymptote. 

Case  (2)  shows  typical  performance  of  the  AT&T  cesium  ensemble.  It  is  monitored  relative  to 
LORAN-C  and  found  to  perform  with  a  long  term  accuracy  within  a  few  parts  in  10-12.  The  daily 
timing  stability  is  limited  by  diurnal  transmission  delay  variation  of  typically  100  ns. 

Case  (3)  shows  the  time  keeping  performance  of  the  rubidium  disciplined  oscillator  assuming 
a  failure  condition  in  which  no  daily  steering  is  applied.  It  is  based  on  a  simplified  assumption 
of  optimal  calibration  of  initial  time,  frequency  and  drift,  and  a  benign  environment.  The  control 
system  applies  feed  forward  correction  for  drift  and  temperature,  and  the  thermal  design  maintains 
small  ambient  temperature  variation  5  °C  daily  objective.  Monte  Carlo  simulation  of  the  holdover 
performance  including  temperature  variation  and  residual  calibration  errors  have  recently  been  done 
which  indicated  that  the  PRC  can  maintain  1  x  1CP11  for  a  period  of  2  weeks  without  steering. 

Cases  (4)  and  (5)  show  the  PRC  performance  under  normal  conditions.  Case  (4)  shows  the 
PRC  three  sigma  time  error  specification  of  200  ns.  The  PRC  three  sigma  performance  bound  is 
based  on  measurements  and  analysis  of  the  PRC  components  and  simulation  of  the  Kalman  based 
steering  algorithm.  Case  (5)  shows  typical  performance  based  on  a  limited  data  recently  obtained  on 
a  prototyped  unit.  The  unit  is  achieving  a  daily  rms  stability  of  25  ns  (3  x  10“ 13). 

It  is  important  to  note  that  even  though  current  standards  allowed  for  plesiochronous  operation 
of  primary  reference  clocks,  all  the  PRCs  in  AT&T  will  be  maintained  to  within  280  ns  (three  sigma) 
of  each  other  with  an  absolute  accuracy  approaching  UTC  (lx  10“ 13  weekly  rms  stability). 
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CONCLUSIONS 


AT&T  ia  deploying  a  GPS  based  system,  termed  a  Primary  Reference  Clock  or  PRC,  which  employs 
a  GPS  receiver  providing  long  term  timing  accuracy  along  with  duplicated  disciplined  rubidium  oscil¬ 
lators  providing  short  term  (ie.  one  day)  stability.  Under  computer  control,  these  three  elements  are 
verified  against  each  other  and  against  identical  PRC  systems  in  other  parts  of  the  network.  For  the 
first  time,  a  national  telecommunications  network  will  be  monitored  and  verified  to  be  performing  to 
a  precision  approaching  UTC. 
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FIG.  2  NEW  VERIFIABLE  DIGITAL  SYNC  PLAN  (REPRESENTATIVE) 
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FIG.  3  PRIMARY  REFERENCE  CLOCK  AND  TIMING  MONITORING  SYSTEM 


INTERBUILDING 

DISTRIBUTION 

FIG.  4  PRIMARY  NODE  TIMING  DISTRIBUTION 
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FIG.  5  SECONDARY  NODE  TIMING  DISTRIBUTION 


GPS  SYSTEM  VERIFICATION  PRIMARY  TIER  VERIFICATION 

(LONG  TERM) 


PRIMARY  REFERENCE  CLOCK  VERIFICATION  SECONDARY  TIER  VERIFICATION 

(SHORT  TERM) 


FIG.  6  LEVELS  OF  VERIFICATION 
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TIME  KEEPIKG  ERROR  vs.  UTC 


QUESTIONS  AND  ANSWERS 

DAVID  ALLAN,  NIST:  Two  questions — first,  the  slope  on  the  red  line  is  not  due  to  a 
frequency  offset  since  it  is  steeper  than  the  frequency  offset.  Is  there  any  explanation  as 
to  why  it  is  that  steep? 

MR.  ZAMPETTI:  That  slope  is  not  due  to  a  frequency  offset.  It  does  converge  to  the 
frequency  offset,  and  I  believe  that  it  is  an  optimal  calibration  estimate,  assuming  random 
walk  FM  as  a  residual.  I  believe  that  that  slope  is  plotted  correctly  as  random  walk 
FM.  That  is  a  theoretical  curve  that  we  just  put  on  for  comparison.  It  shows  that  the 
Rubidium  is  excellent.  We  have  done  some  simulations  incorporating  thermal  instabilities, 
calibration  errors  and  other  things  and  have  found  that  for  a  period  of  two  weeks  the 
system  is  rock  solid  and  would  not  require  any  action.  It  would  still  maintain  the  10'11 
stability  specification. 

MR.  ALLAN:  The  second  question  is:  When  you  steer  the  Rubidium,  do  you  use  C-field 
tuning? 

MR.  ZAMPETTI:  Currently,  that  is  what  you  are  using. 

MR.  ALLAN:  If  that  is  not  done  perfectly,  which  it  never  can,  of  course,  then  it  adds  noise 
to  the  system.  It  is  an  unnecessary  noise  if  one  dealt  with  the  measurements  instead  of 
trying  to  tune  the  C-field. 

MR.  ZAMPETTI:  We  are  very  enthusiastic  supporters  of  synthesis  steering.  As  the  system 
progresses  there  is  the  possibility  of  doing  that.  It  was  not  available  to  us  as  a  matter  of 
practical  course,  but  it  is  something  that  we  are  seriously  considering. 
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